View all comments

Problem

For Ferrocene we would like to run only the coretests and alloctests test suites when measuring code coverage. Running corebenches and allocbenches would alter the numbers, which is not compliant with the certification.

This is currently not possible in bootstrap. By default ./x test runs unit, integration and doc tests. ./x test --doc only runs doc tests. So far, so good.

The problem is that while ./x test --no-doc stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like cargo test --all-targets.

Solution

This PR renames the existing --no-doc flag to --all-targets and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from cargo --bins --examples --tests --benches to cargo --all-targets because it will fail e.g. ./x test library/alloc --all-targets with an error like "use of unstable library feature test".

Additionally, this PR add a ./x test --tests flag (equivalent to cargo test --tests) that only executes unit and integration tests.

Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.

Note that this PR does not change the behaviour of either ./x test or ./x test --doc.

Note on test = true

While this change enables bootstrap to run tests without doc tests and without benchmarks, executing ./x test library/core library/alloc --tests will still build and execute corebenches and allocbenches.

What?! 😱 Why all of this effort to enable it then?

Good question! The reason they are still executed is, that they are marked with test = true in their respective Cargo.toml (corebenches, allocbenches).

@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no ./x --all-targets is passed. This is perfectly possible with this PR. Benchmarks marked with test = true will be executed when calling either of ./x test, ./x test --tests or ./x --all-targets. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.

Open questions

Error message

I added one commit that adds an error message if a user passes --no-doc and points them to --all-targets and --tests. I think it might be nice to have, but if you think it is not necessary, I can remove it.

How to test this change

You can see the change in action by running ./x test library/alloc --tests and ./x test library/alloc --all-targets. The first will execute alloctests and allocbenches (which is marked with test = true), while the second will additionally run benches/vec_deque_append.rs (which is not marked with test = true).